Meta-Classifier Approaches to Reliable Text Classification

A problem with automatic classifiers is that there is no way to know if a particular classification is just a guess or a certain answer. Reliable classification is the task of predicting whether a certain instance is correctly classified or not, i.e., a classification is classified as either reliable or unreliable. When the classification is classified as unreliable, it is like saying “I do not know”, and the instance does not receive a classification. Given a base classifier, the meta-classifier approach is to train a metaclassifier that predicts the correctness of each classification of the base classifier. The classification rule of the meta-classifier approach is to assign a class predicted by the base classifier to an instance if the meta-classifier decides that the base classification is reliable. The meta-classifier approach is applied on text classification tasks provided by the CBS to answer the following problem statement: Does the meta-classifier approach provide a practical solution to reliable text classification? The first part of the research studies text classifiers, and provides an answer to the research question: 1. Which text classifiers achieve high accuracy and at the same time have small space and time complexity? Experiments on the CBS datasets show that the nearest neighbour and the näıve Bayes algorithm in combination with the tfidf text representation are acceptable text classifiers. The second part of the research studies the meta-classifier approach to provide an answer to the second and third research question. Our second research question is: 2. What type of metadata representation is best suited for reliable text classification? The meta-classifier is trained on several types of metadata representations. The used metadata representations include the original instances, the probability distribution of the base classifier and a set of basic statistics about the classification of the base classifier. For tasks with many classes, the original instances representation is best. For tasks with a small number of classes, the original instances representation and the probability distribution are both good candidates.